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ABSTRACT 



This study examines the accuracy of an Army helicopter 
pilot workload measuring model called the Task Loading Model. 
The model is a submodel of the Army-NASA Aircrew/Aircraft 
Integration Program's Man-Machine Integration Design and 
Analysis System. The model's workload level output was 
correlated with the subjective workload measurements of 
several groups of pilots evaluating a variety of flight tasks. 
Seventy-one Army aviators completed surveys requiring scaled 
ratings and paired comparisons of workload related to common 
flight tasks conducted during typical missions. Their 
responses were examined for internal consistency and pooling 
by means of nonparametric tests. Aviator-supplied data was 
found to be robust and reliable. Pooled response data was 
correlated with model-generated data to determine the accuracy 
of the model. Results of this study show that the Task 
Loading Model is presently inadequate, but displays promising 
trends and should be further refined. 
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I. INTRODUCTION 



A. BACKGROUND 

1. Operator Error and the Design Process 

Operator error is the primary cause of many aircraft 
mishaps. "Even if some divine power could guarantee that all 
mechanical and electrical systems used in aviation were 100% 
reliable, there would still be accidents due to human error" 
(Kantowitz, 1988, p.l58). This is especially true in the 
demanding environments associated with military aviation. As 
technological advances raise the performance levels of modern 
military aircraft, the mission employment expectations placed 
upon the aircraft and their operators also rise. As a result, 
pilots encounter more difficult operational environments, 
experience higher workload demands, and freguently surpass 
their human performance capabilities. 

There is a concerted effort among design engineers to 
reduce operator workload through cockpit automation. However, 
a large gap exists between the desired effects of automation 
and its actual ramifications. Instead of lowered operator 
workload, many well intentioned designs result in prolonged 
training, illogical tasking, and excessive demands on pilot 
adaptability. (Army-NASA, 1990, p.2) 
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2. Army-NASA Design Tool: MIDAS 

The U.S. Army and the National Aeronautics and Space 
Administration (NASA) are working together to create a human 
performance model that overcomes the problem of good 
intentions and poor results. The Army-NASA Aircrew/Aircraft 
Integration Program (A^I) is developing a computer simulation 
of aircraft crewstations and their operators. The simulation 
is to serve as a tool that can help ensure that human factors 
considerations are included in system design. 

The Man-Machine Integration Design and Analysis System 
(MIDAS) is one part of the A^I program's efforts. The MIDAS 
workstation presently simulates an AH-64 Apache helicopter, 
its pilot, and the environment in which they operate. MIDAS 
includes a variety of simulation submodels in areas such as 
anthropometry, vision, cockpit display layout, the pilot's 
cognition, and task loading. The synchronization of the MIDAS 
submodels results in a very capable design tool. (Army-NASA, 
1990) 

MIDAS developers envision an interactive computational 
system such as that illustrated in Figure 1. Use of such a 
system will improve the current design process in several 
ways : 

1. It enhances the engineer's efficiency and creativity. He 
or she can configure a cockpit, test it with a human 
model, and immediately evaluate the design's suitability 
in a variety of environments. 
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2. Human factors considerations, often ignored during the 
initial phases of design, are a significant aspect of the 
feedback produced by MIDAS. 

3. Extensive use of computer modeling will reduce the 
expensive development of faulty prototypes, thus reducing 
the research and development costs associated with a 
procurement program. 

4. Although now configured as an Apache helicopter, MIDAS 
will be applicable to any man-machine vehicular system. 
(Army-NASA, 1990, pp.1-6) 




ARMY-NASA AIRCREW/AIRCRAFT INTEGRATION 
PROGRAM (A’l) 



WHAIIS 



Figure l. MIDAS Workstation Concept (Army-NASA, 1990, p.8) 
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3. MIDAS Task Loading Model 



The measurement of pilot workload is an important 
aspect of the aircraft design process. "When workload is 
excessive, errors arise from the inability of the human to 
cope with high information rates imposed by the environment. 
When workload is too low, the human is bored and may not 
attend properly to the task at hand, also leading to error" 
(Kantowitz, 1988, p.l59). 

The Task Loading Model (TLM) is one high resolution 
facet of MIDAS. The TLM manipulates state variables related 
to the operator and the aircraft to calculate the operator's 
relative workload. The TLM output classifies the pilot's 
individual flight tasks and task demands across the visual, 
auditory, cognitive, and motor dimensions. The classifi- 
cations and calculations generate workload data that designers 
may use to determine the level of human performance required 
for a specific system configuration. (Staveland, 1990) 

Four dimensions of human performance are measured by 
the TLM. These four dimensions represent the visual, 
auditory, cognitive and motor activities of a pilot. The TLM 
subdivides the four dimensions into several elements, 
including near and far for a visual activity, salient and 
masked for an auditory activity, planned and unplanned for a 
cognitive activity, and gross and fine for a motor activity. 
Each task is classified according to the four dimensions and 
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the elements within the dimensions. The TLM then generates 
workload values along each dimension. (Staveland, 1990) 

B . TLM PROBLEMS 

Several problems and uncertainties currently are 
associated with the TLM. First, the nature of human workload 
is not fully understood. Although the model incorporates 
"current research in multiple resource theory, scaling, 
workload, and perception" (Army-NASA, 1990, p.l4), limitations 
in the field of behavioral psychology still limit the model. 
Second, current workload measurement mechanisms cannot capture 
all aspects of task loading and human performance. Thus, 
behavioral scientists skeptically view attempts to compile 
workload data. Third, the TLM output requires verification 
with credible workload data to insure that it reflects 
reality. However, even "suspect" helicopter pilot workload 
data suitable for comparison with the TLM output is not 
available . 

There is a critical need to verify the TLM output of 
workload values by comparing them with values obtained by 
other means. This is true despite the present lack of 
standard workload measurement techniques. "The workload 
problems of today will not wait until scientists develop 
perfect models. Fortunately, even incomplete models are 
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useful in developing practical specifications" (Kantowitz, 
1988, p.l82) . 

C. PURPOSE AND SCOPE 

The purpose of this study is to provide data that can be 
correlated with output from the A^I MIDAS TLM. The study 
consisted of three primary procedures. First, subjective 
workload data was collected from subject matter experts 
(helicopter pilots) by means of a survey form. Next, the 
survey data was analyzed to develop a robust, valid set of 
workload values. Finally, TLM output of workload values was 
compared with survey workload data to determine how closely 
the two data sets correlated. 

Chapter II of this thesis discusses the methodology used to 
collect subjective workload estimates from helicopter pilots. 
Chapters III and IV provide the data analysis techniques and 
results. Chapter V outlines conclusions and recommendations. 
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II. METHODOLOGY 



A. STUDY OVERVIEW 

Verifying the reliability of the TLM output was a six-step 
process. The data used for the study came from a survey 
administered to US Army helicopter pilots in a classroom 
environment. This method of data collection allowed for input 
from 71 subject-matter experts in a short time frame and at 
minimal expense. The six steps of the process were: 

1. Identification of helicopter flight tasks for which to 
determine workload values. 

2. Construction of survey forms which were used to collect 
pilot opinions about the workload levels related to these 
tasks . 

3. Determination of data analysis procedures. 

4. Administration of the surveys to Army helicopter pilots. 

5. Analysis of the workload data collected via the survey. 

6. Comparison of the survey data to the TLM output of 
workload values related to the same flight tasks. 

The remainder of this chapter further explains the first four 

steps of the above procedure. 

B. SCEM7VRI0S AMD TASKS 

1. Flight Scenario Development 

Flight tasks for which workload values were collected 
were identified through an analysis of flight missions 
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commonly carried out by the pilots who were surveyed. 
Similar, but not identical, flight scenarios were developed 
both for attack and for scout helicopter pilots, since their 
missions are closely related but vary in significant ways. 
The scenario settings used for the survey are provided in the 
survey forms, examples of which are included as Appendix A 
(scout helicopter) and Appendix B (attack helicopter) . 

Each of the two scenario descriptions was developed by 
a pilot with experience in both the mission and the type of 
aircraft "flown" in the scenario. Each scenario represents 
approximately a two-minute slice of time from a typical attack 
or reconnaissance mission. Tasks occurring during the 
scenario are based on what a realistic cockpit crew is 
expected to do, under the assumption that the survey 
respondent is manipulating the flight controls. This creates 
a single-pilot situation from which consistent task workload 
measurements could be drawn. 

2. Task Selection 

The specific tasks associated with each scenario were 
next identified. Of the more than 20 maneuvers conducted by 
the pilot throughout the scenario, nine items were selected as 
"individual tasks," which are defined as discrete activities 
carried out by a pilot while flying (see Tables 1 and 2). An 
activity is a physical or mental process associated with a 
helicopter flight mission. Activities may be described by the 
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four dimensions of human performance associated with the TLM: 
the visual, auditory, cognitive and motor dimensions. 

Additionally, the nine individual tasks were merged 
into six "combined tasks," also listed in Tables 1 and 2. 
Combined tasks consist of discrete events that overlap along 
a mission time line. For example. Tell crewmember unmasking 
is one individual task and Unmask aircraft is another individ- 
ual task. However, Tell crewmember "Unmasking" while unmask- 
ing the aircraft is a combined task. 

Since workload levels collected via the surveys were 
to be compared with those generated by the MIDAS TLM, it was 
important that the tasks included in the survey forms consist 
of a potentially wide variety of workload levels. The nine 
individual tasks selected from the two scenarios met this 
requirement. As a whole, they encompassed the range of 
loading evaluated by the TLM. They were selected intentional- 
ly to verify all dimensions of the model, as described in 
Chapter I. 

The nine individual tasks selected for the attack and 
scout helicopter pilots were exactly the same or parallel. 
For example. Task 4 for both groups of pilots was Hover. 
However, Task 8 for the attack helicopter pilots was Switch 
master arm to "arm," while for the scout pilots it was Turn up 
volume on ICS control panel . Both tasks involve reaching with 
the left hand and adjusting a switch or knob with the fingers. 
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TABLE 1 



ATTACK HELICOPTER FLIGHT TASKS 



TASK 

No. 


TASK 

CODE 


INDIVIDUAL TASKS 


4 


Folw 


Follow scout 


t 


Decl 


NOE deceleration 


3 


Cktq 


Check torque meter and announce reading 


4 


Hovr 


Hover 


5 


Tell 


Tell crewmember "Unmasking” 


6 


Umsk 


Unmask aircraft 


7 


Lstn 


Listen to radio transmission from scout 
(direction to fire, range, target) 


8 


Swch 


Switch master arm to "arm" 


9 


Intp 


Interpret gunner's position of TSU based on 
PSI information 


COMBINED TASKS 


1 


Cdecl 


NOE deceleration behind scout 


2 


Ccktq 


Check torque meter and announce reading 
while NOE decelerating 


3 


Cckhv 


Check torque meter and announce reading 
while hovering 


4 


Cumsk 


Tell crewmember "Unmasking" while unmasking 
aircraft 


5 


Clstn 


Listen to radio transmission from scout 
(direction to fire, range, target) while i 

switching master arm to "arm" 


6 


Cintp 


Interpret gunner's position of TSU, based 
on PSI information, while receiving target 
handoff from scout 
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TABLE 2. SCOUT HELICOPTER FLIGHT TASKS 



TASK 


TASK 




No. 


CODE 


INDIVIDUAL TASKS 


t 


Folw 


Follow another scout 


2 


Decl 


NOE deceleration 


3 


Cktq 


Check torque meter and announce reading 


4 


Hovr 


Hover 


5 


Tell 


Tell crewchief "Unmasking” 


6 


Umsk 


Unmask aircraft 


7 


Lstn 


Listen to crewchief describe what he sees 


8 


Turn 


Turn up volume on ICS control panel 


9 


Detm 


Determine present location (6 digit grid) 
as crewchief holds map 


COMBINED TASKS 


4 


Cdecl 


NOE deceleration behind another scout 


2 


Ccktq 


Check torque meter and announce reading 
while NOE decelerating 


3 


Cckhv 


Check torque meter and announce reading 
while hovering 


4 


Cumsk 


Tell crewchief "Unmasking" while unmasking 
aircraft 


5 


Clstn 


Listen to crewchief describe what he sees 
while you turn up volume on ICS control 
panel and hover 


6 


Cdetm 


Determine present location (6 digit grid) 
as crewchief holds map and you listen to 
him describe what he sees 
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C. SUBJECTIVE WORKLOAD MEASUREMENT TECHNIQUES 

1. Selection of Measurement Techniques 

Several means of eliciting workload judgements from 
pilots were considered. The desired characteristics of a 
workload measuring mechanism were as follows: 

1. Reliable and credible results are obtained. 

2. An appropriate level of effort is required of the 

respondent. 

3. The resulting subjective measurements can be quantified. 

4. Administration to groups of 15 or more is easy. 

5. Data analysis is straightforward. 

Two established techniques were chosen for this study, 
since both met the above requirements. They were subjective 
workload rating scales and the psychometric technique known as 
Thurstone's Method of Paired Comparisons (ARI Report 851, 
1989, pp. 81-122). Examples of both of these techniques are 
included in the example survey forms located in Appendix A and 
Appendix B. 

2 . Rating Scales 

Workload rating scales require the respondents to 
assign a numerical value (from 0 to 10 for this study) to each 
task, thereby indicating its level of difficulty. The use of 
rating scales was advocated by the designer of the MIDAS TLM, 
resulting from his experience with the NASA-Task Load Index 
(TLX) (Hart & Staveland, 1987) . The TLX is a highly refined 
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subjective workload assessment scaling technique based on 
rating scale methodologies. Rating scales meet the five 
requirements listed above for a workload measuring mechanism. 
They allow for any number of subjects, tolerate missing 
responses, and facilitate efficient statistical analysis. A 
potential drawback is that raters become frustrated when 
required to make large numbers of judgements. Thus, reducing 
possible frustration was a major reason for including only 
nine tasks on the survey form (Zatkin, 1983) . 

3. Thurstone's Method 

Thurstone's method utilizes several judges who compare 
each item in a set against each of the other items, one at a 
time. This method results in an ordering of the items 
according to a relative value of magnitude interdependent on 
every other compared item. For the workload measurements, 
each pilot determined which of two compared tasks was more 
difficult. Judgements were made for each set of two until all 
individual or combined tasks had been compared with all the 
other individual or combined tasks. From these comparisons a 
scale of relative task difficulty emerged. (Thurstone, 1963) 

A drawback to Thurstone's Method is the potentially 
cumbersome number of pairwise comparisons required of the 
respondent. To compare n items against each other requires 
n(n-l) /2 judgements. Workload measurements for this study 
required 36 comparisons of the nine individual tasks and 15 
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comparisons of the six combined tasks. This number of 
pairwise comparisons and the accompanying scale ratings 
provided a challenging, but not overwhelming endeavor for the 
pilots participating in the survey. 

D. SURVEY CONSTRUCTION 

1. Survey format 

Two separate survey forms were used to measure 
helicopter pilot experience with workload during flight, one 
for scout helicopter pilots and the other for attack helicop- 
ter pilots (see Appendices A and B) . Each was designed to be 
completed in approximately 15-20 minutes. The 12-page survey 
forms included an introduction, instructions on how the survey 
should be filled out, and the scenario setting. The survey 
itself was divided into four sections: 

1. Individual task subjective rating scales. 

2. Combined task subjective rating scales. 

3. Individual task pairwise comparisons. 

4. Combined task pairwise comparisons. 

Pilot background information also was collected. 

Ratings and comparisons were randomized within each 
section for each individual survey form. The four sections 
also were shuffled into their 24 different permutations, to 
create 24 unique formats. Rosenthal points out that "very 
large questionnaires can tend to discourage the subjects from 
answering. A way around this problem is to vary the format in 
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order to keep the subjects interested" (Rosenthal, 1984, 
p.l34). Besides maintaining subject interest, randomization 
also reduced the chances that judges might respond according 
to a set contextual flow of the survey material (Rosenthal, 
1984) . 

2 . Survey Content 

The first page of each survey form provided background 
information to the pilots. Several points were highlighted: 

1. The survey's purpose: attain workload information to aid 
in the improvement of future Army helicopter cockpits. 

2. Emphasis on candid responses, with no right or wrong 
answers . 

3. Individual anonymity and the pooling of response data. 

4. The uniqueness of each survey form due to question 
randomizing. 

These points were discussed to help control what Rosenthal 
calls "demand characteristic: the subject's perception of his 
or her role and of the experimenter's hypothesis." Clarifying 
the pilots' role made them more sensitive to the aims of the 
survey, thereby motivating a response, while instilling 
confidence in their ability to respond appropriately. 
(Rosenthal, 1984, p.l05) 

The second page of the form consisted of detailed 
instructions indicating how to respond. Specific, relevant 
examples were included. This was followed by a page contain- 
ing a one-paragraph scenario description providing the context 
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within which the pilot was to respond. The setting was that 
of a typical tactical flight mission. Applicable parameters 
such as crew mix, weather conditions, and the enemy situation 
were described. The forms for the actual ratings and compari- 
sons followed. The final page contained a short questionnaire 
regarding the aviation experiences and qualifications of the 
responding pilot. 

The two survey forms were reviewed for accuracy of 
content and wording by two Army instructor pilots. They 
provided helpful feedback by recommending current aviation 
phraseology, such as changing "copilot" to "crewmember." They 
verified the sound structure of the scenarios and reported 
that the survey was appropriate for the objectives that were 
to be met. 

3. Minimizing the Social-Desirability Response 

A concept from the field of behavioral science that 
influenced survey construction was what is referred to as the 
"social-desirability response set" (Rosenthal, 1984, pp.l35- 
139). With respect to helicopter pilots, this mindset would 
tend to cause them to tailor their responses so they would 
appear to be competent pilots who rarely experience demanding 
workload. Three techniques were utilized to minimize the "I 
can handle anything" mentality common among pilots. First, 
anonymity was emphasized so that no individual could be linked 
to a specific response. Second, the forced choices associated 
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with Thurstons' s paired comparisons required the respondent to 
specify that one task was harder than another. This prevented 
respondents from claiming that none were difficult. The third 
technique involved the interaction of the survey administrator 
with the pilots who were completing the survey, as discussed 
in the next section. 

E. SURVEY ADMINISTRATION 
1. Survey Respondents 

Surveys were administered to pilots assigned to the 
Seventh Infantry Division (Light) located at Fort Ord, 
California. Pilots from the 2/9 Cavalry Squadron completed 
the survey on 29 March 1991. Pilots from the 1/123 Attack 
Helicopter Battalion completed the survey on 23 May 1991. 
Table 3 shows the number of respondents by category. 

A third group of respondents consisted of Enlisted 
Aerial Observers from the same Squadron and Battalion, 
surveyed on the same dates. Aerial Observers are enlisted 
soldiers who accompany scout pilots during a mission. 
Although their tasks do not usually include hands-on flying, 
each has at least 60 hours of flight training prior to 
reporting to an aviation unit. As flight crewmembers working 
in the cockpit, they have experience with the surveyed tasks. 
Thus, their responses contributed a third pool of information 
to the data set. 
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TABLE 3. SURVEY RESPONDENTS 





2/9 Cavalry 
Sqdn 


1/123 
Attack Bn 


Total by 
Group 


Attack Helicopter 
Pilots 


13 


26 


39 


Scout Helicopter 
Pilots 


12 


6 


18 


Aerial Observers 


8 


6 


14 


Total by Unit 


33 


38 


71 



2 . Survey Administration Procedures 

The surveys were administered in a classroom environ- 
ment during regularly scheduled pilot meetings. Before 
receiving the questionnaire, the pilots and observers were 
given a short briefing by the survey administrator, an Army 
captain. He used an overhead projector to provide necessary 
information to the subjects and emphasized the following: 

1. The possible effect of their efforts on future aircraft 
design. 

2. An overview of the A^I and MIDAS systems. 

3 . A short discussion of human factors and its role in 
aircraft design. 

4. Examples of survey questions and responses. 

At the end of the briefing, the survey forms were distributed 
to the respondents by each unit's instructor pilots. All 
forms were completed within 15-25 minutes after distribution. 
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III. ANALYSIS OF STUDY DATA 



A. DATA ANALYSIS SCHEME 

The data collected from the 71 subjects surveyed 
analyzed according to the scheme depicted in Figure 2. 




concepts guiding the analysis were: 

1. Determine the degree of homogeneity among 
respondents. 



was 

The 



the 
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2. Attain workload values for the tasks evaluated by the six 
groups of subjects from a Cavalry Squadron and an Attack 
Helicopter Battalion, each consisting of attack 
helicopter pilots, scout helicopter pilots, and aerial 
observers. 

3. Filter the groups of data by means of nonparametric 
statistical tests. 

4. Accept a final pool of workload measurements that meet 
the above criteria. 

These concepts were utilized to provide an internally 
consistent set of study data. 

B. PILOT BACKGROUND INFORMATION 

The last page of the workload survey consisted of 
questions intended to establish each subject's aviation 
background and experience (see Appendix A or B) . The pilot 
background data was graphically analyzed using Box plots 
(Rice, 1988, pp. 336-337) in order to compare the aviators in 
the 2/9 Cavalry Squadron with those of the 1/123 Attack 
Helicopter Battalion. Three particular measures were 
analyzed: 

1. Total flight time. 

2. Flight time in the aviator's primary aircraft: either the 
attack helicopter or the observation helicopter. 

3. Flight time as a pilot-in-command. 

Figure 3 shows a multiple Box plot of the total flight 
time of all the subjects. The first plot represents the 
attack helicopter pilots of the Attack Battalion (AH-AB) 
alongside those of the Cavalry Squadron (AH-CV) in the second 
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plot. The third and fourth plots (SH-AB and SH-CV) are those 
of the Scout pilots. The two groups of observers (EO-AB and 
EO-CV) are plotted last. 




Figure 3. Multiple Box Plots of Grouped Aviator 
Total Flight Time 



These Box plots provide an efficient method of analysis 
for several groups of data. Several observations can be made, 
based on the complete set of Box plots analyzed during this 
study. These observations include the following: 

1. When grouped within their respective units, the attack 
helicopter pilots of the Cavalry Squadron and Attack 
Battalion have very similar backgrounds in total flight 
time, attack helicopter flight time, and pilot-in-command 
flight time. 

2. There was one attack helicopter pilot outlier, a pilot in 
the Cavalry Squadron with almost 3000 hours of total 
flight time. 
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3. The grouped scout helicopter pilot data indicated that 
subjects from the Cavalry Squadron averaged approximately 
100 hours more experience in the three flight time 
categories than those from the Attack Battalion. 

4 . Each scout helicopter unit had one outlier with slightly 
more than 2000 hours total flight time. 

5. Only total flight time data was analyzed for the enlisted 
aerial observers. Box plots for observers from both 
units are almost identical. 

C. DATA TRAKSFORMATIOM AND SUMMARIZATION 

The data collected on the survey forms was transformed 
into 24 response vectors of either six or nine values. Each 
vector was derived from grouped subject responses for either 
the individual or combined task data from the rating scales or 
paired-comparisons. In other words, these 24 vectors 
represented responses from the six aviator groups for the four 
sections of the survey. 

The rating scale responses were transferred from each 
survey form to a spreadsheet. There they were recorded, 

summed, and averaged. The resultant values are listed in 
Table 4, where smaller values denote lower workload levels. 
The paired comparison responses were transferred from the 
survey forms into pooled groups of workload values by means of 
Thurstone's Method (Thurstone, 1963, pp. 67-81). The 
particular method used, Thurstone's Case V (Dunn-Rankin, 1983, 
pp. 79-82) , resulted in the values listed in Table 5. 
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TABLE 4. INITIAL AVERAGE TASK RATING WORKLOAD VALUES 



INDIVIDUAL TASKS 




Attack Pilots 


Scout Pilots 


Observers 


Group 

Code 


AH-AB 


AH-CV 


SH-AB 


SH-CV 


EO-AB 


EO-CV 


Task 

No. 


Attack 

Bn 


Cav 

Sqdn 


Attack 

Bn 


Cav 

Sqdn 


Attack 

Bn 


Cav 

Sqdn 


1 


3.87 


4 . 19 


2.17 


2.08 


2.83 


4 . 19 


2 


3.40 


3.46 


2.50 


2.75 


3 . 83 


4 . 69 


3 


2.33 


2.38 


1.17 


1.58 


2.83 


2 . 19 


4 


2.56 


2.81 


2.17 


1.83 


2.17 


2.81 


5 


1. 13 


0.88 


0.67 


0.83 


0.58 


1.31 


6 


2.67 


3.77 


1.83 


2.25 


2.33 


3.06 


7 


3.29 


3.38 


1.00 


2.42 


1.83 


2.19 


8 


2.06 


2.58 


2.33 


2.00 


1.33 


1.56 


9 


3.42 


2.92 


5.50 


5.92 


4.50 


6.56 


COMBINED TASKS 


1 


2.33 


4.85 


4.29 


3.58 


4.83 


4.69 


2 


3.83 


4.85 


4 . 17 


4.75 


5.50 


5.56 


3 


1.33 


2.62 


2.63 


1.42 


3 . 50 


1.69 


4 


0.50 


1.62 


2.10 


1.33 


1.67 


1.56 


5 


3 . 17 


4 . 00 


4 . 13 


3.08 


4.33 


3.19 


6 


5.50 


4 . 35 


4.94 


5.58 


5.83 


5.44 


n 


26 


13 


6 


12 


6 


8 
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TABLE 5 



INITIAL AVERAGE PAIRED-COMPARISON WORKLOAD 
VALUES 



INDIVIDUAL TASKS 




Attack Pilots 


Scout Pilots 


Observers 


Group 

Code 


AH-AB 


AH-CV 


SH-AB 


SH-CV 


EO-AB 


EO-CV 


Task 

No. 


Attack 

Bn 


Cav 

Sqdn 


Attack 

Bn 


Cav 

Sqdn 


Attack 

Bn 


Cav 

Sqdn 


1 


1.93 


2.77 


2.90 


1.25 


1.97 


1.47 


2 


1.82 


2.45 


2.90 


1.59 


2.44 


1.66 


3 


1.13 


0.87 


1.40 


0.27 


1.78 


0.55 


4 


1.41 


1.14 


1.91 


0.89 


1.72 


0.57 


5 


0.00 


0.00 


0.00 


0.00 


0.00 


0.00 


6 


1.59 


1.76 


2.28 


1.19 


1.88 


0.46 


7 


1.68 


2 . 14 


1.19 


0.93 


0.93 


0.31 


8 


1.23 


1.68 


2.06 


0.91 


1.03 


0.54 


9 


1.54 


1.77 


3.99 


2.82 


2.74 


2.42 


COMBINED TASKS 


1 


1.51 


1.75 


2.48 


1.87 


1.08 


1.08 


2 


2.08 


2.12 


2.49 


1.95 


2.80 


1.39 


3 


0.53 


0.77 


1.00 


0.30 


0.78 


0.00 


4 


0.00 


0.00 


0.00 


0.00 


0.00 


0.09 


5 


1.39 


2.18 


2.26 


1.72 


0.85 


1.39 


6 


1.75 


2.60 


3.43 


2.33 


2.88 


1.89 


n 


26 


13 


6 


12 


6 


8 
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The workload values listed in Tables 4 and 5 are 



characterized as follows: 

1. Group codes, identifying columns of workload values, 
correspond with the similarly-labelled graphic data for 
the Squadron and Battalion used throughout this study 
(See Figure 3) . 

2. Tasks are listed in order of their occurrence during the 
flight scenario. 

3. The greater the workload value, the more difficult the 
task. 

4. Thurstone's method assigns the easiest task the value 
zero. Higher-valued tasks are assigned a quantitative 
workload measure according to their relative difficulty 
in relation to the other tasks. 

5. The bottom row in each table indicates the number of 
respondents, n, whose responses were used to calculate 
each column's workload values. 

These 24 vectors of workload data provide the input for the 
subsequent analysis and tests to follow. 

D. PLOTS OF WORKLOAD DISTRIBUTION 

Plots of the workload data in Tables 4 and 5 provided a 
quick graphic assessment of the distribution of workload 
according to task (Chambers, 1983, pp. 82-86). Most of the 
plots constructed for the study compared the responses of one 
group of aviators from the Cavalry Squadron with those in the 
same type of group from the Attack Helicopter Battalion. 
Three representative graphs are depicted in Figures 4, 5, and 
6 . 

Figure 4 shows the plot of individual-task paired- 
comparison workload values for the attack helicopter pilots in 
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both the Cavalry Squadron and the Attack Helicopter Battalion. 
Figure 5 shows combined-task paired-comparison workload data 
for the scout pilots of both units. Figure 6 shows a plot of 
all of the individual task ratings provided by all subjects, 




Figure 4. Plot of Attack Helicopter Pilot Workload 
Measurements Based on Individual-Task Paired 
Comparisons 




COMBINED TASKS 



Figure 5. Plot of Scout Helicopter Pilot Workload 
Measurements Based on Combined-Task Paired 
Comparisons 
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averaged according to their six respective aviator and unit 
groups. All three plots indicate that there is a general 
trend toward like responses among the groups of aviators. 




Figure 6. Plot of Aviator Workload Measurements Based on 
Individual Task Ratings by All Study 
Participants 



E. NONPARAMETRIC TESTS 

Several nonparametric statistical tests of the data were 
employed to investigate possible trends and to determine if 
responses were consistent among the subjects. These tests 
were used to explore the validity of combining the workload 
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data vectors into fewer pools of data, with more respondents 
contributing to each pool. 

1. Friedman Test 

The Friedman test for randomized block designs 
(Mendenhall, 1990, pp. 702-704) was used to determine if the 
responses of each pilot grouping came from identical 
distributions. The significance level for this test was set 
at 0.05. Thus, if the groups tested attained a p value 
greater than 0.05, then they would be considered to come from 
the same underlying distribution. 

Friedman tests were conducted on data for each of the 
four sections of the survey. Individual- and combined-task 
ratings and paired comparisons for each of the six aviator 
groups were tested as randomized blocks. Figure 7 graphically 
portrays the Friedman Test results for one section of the 
survey, the individual-task paired comparisons. The varying 
bar lengths indicate that the six groupings appear to differ 
in the average of the ranks given to workload values. This is 
numerically supported by the p value of 0.0001 obtained from 
the Friedman test for this randomized block. 

Similar results were found for initial average 
workload values from the other three survey sections. Table 
6 shows the p values for data from all four sections. Thus, 
for each of the four sections, it cannot be assumed that all 
six groups of respondents are from the same underlying 
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distribution. Therefore, all six response vectors should not 
be combined into one data pool for any of the four survey 
sections, as none of the p values exceeded 0.05. 



AH-AB 

AH-CV 

SH-AB 

SH-CV 

EO-AB 

EO-CV 



LE 



0 






3 



2 3 4 

AVERAGE RANK 






Figure 7 . Friedman Test Average Ranks of Aviator Groups 
for the Individual Task Paired-Comparisons 



TABLE 6. FRIEDMAN TEST P VALUES 



SECTION 


P VALUE 


IMPLICATION 


Individual Task Ratings 


0.0071 


Do not pool data from 
top half TABLE 4 


Combined Task Ratings 


0.0114 


Do not pool data from 
bottom half TABLE 4 


Individual Task Paired 
Comparisons 


0.0001 


Do not pool data from 
top half TABLE 5 


Combined Task Paired 
Comparisons 


0.0448 


Do not pool data from 
bottom half TABLE 5 
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2 . Spearman Rank Correlations 

Since the Friedman test indicated that the data could 
not be combined into one data pool, Spearman rank correlations 
(Hamburg, 1987, pp. 576-578) were utilized to determine if 
data from any of the four sections could be pooled for any of 
the six aviator groups. Comparisons were made between like 
aviators of the Attack Battalion and the Cavalry Squadron. 
For example, the responses of the attack pilots of the Attack 
Helicopter Battalion (AH-AB) were correlated with those of the 
Cavalry Squadron (AH-CV) attack pilots. The significance 
level of 0.006 (shown in the upper left portion of Table 7) 
indicates a high degree of correlation between these two 
groups of responses. 

Of the groups of aviators, those that correlated with 
a significance level less than 0.05 were considered to be 
candidates for data pooling. Table 7 shows the significance 
levels associated with the Spearman rank correlations of the 
six aviator groups for the individual-task paired comparisons. 
Only the correlations of like groups (those performing similar 
missions) are included. Additionally, scout pilot and 
enlisted observer comparisons were only made when the two 
groups were in the same unit. For instance, the responses of 
the Cavalry Squadron Observers (EO-CV) were correlated only 
with Cavalry Squadron scout pilots (SH-CV) , not with Attack 
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Battalion scouts (SH-AB) . However, cross unit observer 
comparisons were made (EO-AB with EO-CV) . 

The results of the Spearman rank correlations for 
individual-task paired comparisons are more favorable to data 
pooling than are results obtained for data from the other 
survey sections (significance levels for the Spearman rank 
correlations of the other three sections of the survey are 
located in Appendix C) . The bold numbers indicate groups 
whose correlations are statistically significant (i.e., less 
than 0.05) . 



TABLE 7. RANK CORRELATION SIGNIFICANCE LEVELS 



INDIVIDUAL-TASK PAIRED COMPARISONS 




AH-AB 


AH-CV 


SH-AB 


SH-CV 


EO-AB 


EO-CV 


AH-AB 


- 


0.006 


- 


- 


- 


- 


AH-CV 




- 


- 


- 


- 


- 


SH-AB 






- 


0.011 


0.009 


- 


SH-CV 








- 


- 


0.048 


EO-AB 










- 


0.013 


EO-CV 












- 



Figure 8 is a graph of the Spearman rank correlations 
for the five comparisons of data from the individual-task 
paired comparisons where correlations are statistically 
acceptable for pooling (AH-CV and AH-AB, SH-CV and SH-AB, SH- 



31 



AB and EO-AB, SH-CV and EO-CV, and EO-AB and EO-CV) . These 
five comparisons correspond with the five bold significance 
levels in Table 7. The Spearman test results (low 
significance levels and high correlations) for these five 
group comparisons indicates similar underlying distributions 
for each pair and suggests that pooling may be appropriate. 




Figure 8. Spearman RanX Correlations for the Individual 
Task Paired Comparisons 



3. Wilcoxon Signed RanX Test 

Before accepting the groupings of data suggested by 
the Spearman test, a final nonparametric test was conducted. 
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The Wilcoxon signed rank test (Mendenhall, 1990 , pp. 680 - 681 ) 
provided a method of determining the sameness or difference of 
the aviator workload response distributions. The same aviator 
group comparisons made during the Spearman test were made with 
the Wilcoxon test. The p values generated by the Wilcoxon 
test are listed in Table 8 for the task ratings and in Table 
9 for the paired comparisons. P values greater than a 
significance level of 0.05 imply that the underlying 
distributions are similar. Values in this category are printed 
in bold type. 



TABLE 8. WILCOXON SIGNED RANK TEST RESULTS FOR THE TASK 
RATINGS OF CORRELATED GROUPS 





INDIVIDUAL TASKS 


COMBINED TASKS 


Compared Groups 


No . of 
Pairs 


p value 


No. of 
pairs 


p value 


AH-AB & AH-CV 


9 


0.17 


6 


0.21 


SH-AB & SH-CV 


9 


0.16 


6 


0.14 


EO-AB & EO-CV 


9 


0.03 


6 


0.06 


SH-AB & EO-AB 


9 


0.44 


6 


0.09 


SH-CV & EO-CV 


9 


0.02 


6 


0.09 



Tables 9 and 10 show the following: 



1. Workload values may be pooled for both the individual- 
and combined-task ratings for AH-AB & AH-CV, SH-AB & SH- 
CV, and SH-AB & EO-AB. 
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TABLE 9 



WILCOXON SIGNED RANK TEST RESULTS FOR THE 
PAIRED COMPARISONS OF CORRELATED GROUPS 





INDIVIDUAL TASKS 


COMBINED TASKS 


Attack Bn/Cav Sqdn 
Samples 


No. Of 
Pairs 


p value 


No. of 
pairs 


p value 


AH-AB & AH-CV 


9 


0.14 


6 


0.06 


SH-AB & SH-CV 


9 


0.01 


6 


0.06 


EO-AB & EO-CV 


9 


0.01 


6 


0.28 


SH-AB & EO-AB 


9 


0.04 


6 


0.18 


SH-CV & EO-CV 


9 


0.11 


6 


0.06 



2. Workload values may be pooled for only the combined-task 
ratings for EO-AB & EO-CV and SH-CV & EO-CV. 

3. Workload values may be pooled for both the individual- 
and combined-task paired comparisons for AH-AB & AH-CV 
and SH-CV & EO-CV. 

4. Workload values may be pooled for only the combined-task 
paired comparisons for SH-AB & SH-CV, EO-AB & EO-CV, and 
SH-AB & EO-AB. 

Results obtained from the Wilcoxon signed rank test indicate 
that other values cannot be pooled. The test showed that the 
underlying distributions were not the same. 

F. POOLED WORKLOAD RESPONSES 

Pooled workload values resulting from the filtering effect 
of the nonparametric tests are listed in Table 10 for the task 
ratings and Table 11 for the paired comparisons. The data in 
Tables 10 and 11 represents the highest level of pooling that 
can be justified by the nonparametric tests. In other words, 
data from the groups of aviators was pooled as shown in Tables 
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10 and 11 if the Spearman test and the Wilcoxon test signifi- 
cance level values are shown in bold type in both Table 7 (or 
Appendix C) and Table 8 or 9 . 

The values listed in Tables 10 and 11 were generated by 
recomputing the raw data from the survey forms. The 
recomputations involved transforming the pooled aviator 
response data into appropriate task ratings and Thurstone 
paired-comparison values. This process was carried out only 
for data groups shown in Tables 4 and 5 which were highly 
correlated according to the Spearman test, and were determined 
to be of the same underlying distribution by the Wilcoxon 
test. Data that did not meet these requirements was not 
included in Tables 10 and 11. Thus, four columns in Table 10 
and four columns in Table 11 are blank. 

Absent from Tables 10 and 11 are pooled workload values 
for the enlisted aerial observers (EO-AH and EO-CV) . As these 
aviators are not qualified pilots, their responses considered 
alone may not accurately portray workload. Therefore, the 
observers' responses were included in Tables 10 and 11 only 
when they agreed with those of the scout pilots, according to 
the Spearman and Wilcoxon tests. The pooled workload values 
of the two right hand columns of Tables 10 and 11 include 
responses from the enlisted aerial observers pooled with those 
of the scout pilots. 
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TABLE 10. POOLED AVERAGE TASK RATING WORKLOAD VALUES 



INDIVIDUAL TASKS 


Group 

Code 


AHR 


SHR 


SOR-AB 


SOR-CV 


Task 

No. 


All 

Attack 

Pilots 


All 

Scout 

Pilots 


Attack Bn 
Scout & 
Obser 


Cavalry 
Sqdn Scout 
& Obser 


1 


3.97 


Cannot be 
pooled 
according 
to 

Spearman 

Test 


Cannot be 
pooled 
according 
to 

Spearman 

Test 


Cannot be 
pooled 
according 
to 

Wilcoxon 

Test 


2 


3.42 


3 


2 .35 


4 


2.64 


5 


1.05 


6 


3.04 


7 


3.32 


8 


2.23 


9 


3.26 




COMBINED 1 


’ASKS 




1 


Cannot be 
pooled 
according 
to 

Spearman 

Test 


3.17 


3.58 


4.03 


2 


4.44 


4.67 


5.08 


3 


1.39 


2.42 


1.52 


4 


1.06 


1.08 


1.43 


5 


3.11 


3.75 


3.13 


6 


5.56 


5.67 


5.53 


n 


39 


18 


12 


20 
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TABLE 11. POOLED AVERAGE PAIRED-COMPARISON WORKLOAD 
VALUES 



INDIVIDUAL TASKS 


Group 

Code 


AHP 


SHP 


SOP-AB 


SOP-CV 


Task 

No. 


All 

Attack 

Pilots 


All 

Scout 

Pilots 


Attack Bn 
Scout & 
Obser 


Cavalry 
Sqdn Scout 
& Obser 


1 


2 . 03 


Cannot be pooled 
according to 
Wilcoxon 
test 


1.32 


2 


1.87 


1.58 


3 


0.96 


0.41 


4 


1.28 


0.78 


5 


0.00 


0.00 


6 


1.55 


0.94 


7 


1.69 


0.69 


8 


1.28 


0.81 


9 


1.50 


2 . 64 


COMBINED TASKS 


1 


Cannot be 
pooled 
according 
to 

Spearman 

test 


1.99 


0.93 


Cannot be 
pooled 
according 
to 

Spearman 

test 


2 


1.99 


1.02 


3 


0.50 


0.07 


4 


0.00 


0.00 


5 


1.72 


0.94 


6 


2.51 


1.35 


n 


39 


18 


12 


20 
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G. CORRELATION OF RATINGS WITH PAIRED-COMPARISONS 

Spearman's rank correlations were used with the pooled 
data of Tables 10 and 11 to verify the internal consistency of 
aviator responses. Workload ratings for the individual and 
combined tasks were compared with the paired-comparison 
workload values for the same tasks for those cases where both 
Table 10 and Table 11 had data in the same columns. 

For the individual tasks the only correlation that met 
this requirement was that for the attack helicopter pilots. 
The significance level of 0.007 for the Spearman correlation 
of all attack helicopter pilot individual-task workload 
ratings (AHR, Table 10) with related paired-comparison 
workload values (AHP, Table 11) indicates consistent attack 
helicopter pilot responses throughout the survey for these 
tasks. 

Combined-task workload values from Tables 10 and 11 also 
were tested. Table 12 provides the significance levels found 
for the Spearman correlations. Only scout helicopter pilot 
and enlisted aerial observer response data are included, as 
the attack helicopter pilot responses could not be pooled. 
All values in Table 12 are below the 0.05 significance level, 
suggesting consistency in the scout and observer responses for 
the combined tasks. 
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Results of the nonparametric tests prevented pooling of 
scout helicopter pilot individual task responses for the 
Attack Helicopter Battalion and the Cavalry Squadron. Thus 
Spearman correlations were performed on the initial workload 
values for the individual tasks of the scout helicopter pilots 
(Tables 4 and 5). Attack Battalion scout pilots' individual- 
task ratings (SH-AB, Table 4) and paired-comparisons (SH-AB, 
Table 5) correlated with a significance level of 0.013. 
Cavalry Squadron scout pilots' individual-task ratings (SH-CV, 
Table 4) and paired comparisons (SH-CV, Table 5) correlated 
with a significance level of 0.008. As with the pooled 
observations above, the significance levels for correlation 
between the two kinds of initial workload values indicates 
consistency among the aviators' responses. 

TABLE 12 . RAKK CORRELATION SIGNIFICANCE LEVELS FOR POOLED 
WORKLOAD VALUES 



COMBINED TASKS: RATINGS AND PAIRED COMPARISONS 





SHR 


SOR-AB 


SOR-CV 


SHP 


SOP-AB \ 


SHR 


- 


0.035 


0.035 


0.028 


0.035 1 


SOR-AB 




- 


0.035 


0.045 


0.000 


SOR-CV 






- 


0.028 


0.035 


SHP 








- 


0.045 


SOP-AB 










- 
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Nonparametric test results also prevented pooling of 
attack helicopter pilot combined-task responses for the Attack 
Helicopter Battalion and the Cavalry Squadron. Thus, as with 
the scout pilot comparisons above, Spearman correlations were 
performed on the initial workload values (Tables 4 and 5) . 
Attack Battalion attack pilots' combined-task ratings (AH-AB, 
Table 4) and paired comparisons (AH-AB, Table 5) correlated 
with a significance level of 0.048. Cavalry Squadron attack 
pilots' combined-task ratings (AH-CV, Table 4) and paired 
comparisons (AH-CV, Table 5) correlated with a significance 
level of 0.243. From these correlations, the Attack Battalion 
attack pilots' combined-task workload values are consistent 
for the ratings and the paired comparisons. However, the high 
significance level (0.243) for the Cavalry Squadron does not 
support consistency between the responses to the ratings and 
the paired comparisons. 

For the correlations of the task rating workload values 
with the paired-comparison workload values, low significance 
levels indicate high correlations. The results discussed 
above show that the subjects' responses were consistent 
throughout the survey. Furthermore, test results confirm 
robustness of the data and a high degree of reliability among 
the selected groups of responses. Based on these tests, the 
data in Tables 10 and 11 and selected values of Tables 4 and 
5 (except for Cavalry Squadron attack helicopter pilot 
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responses) can be confidently used as baseline workload values 
for the tasks and conditions considered. These baseline, 
empirically-determined values serve as "ground truth" data 
that can be used to validate the Midas Task Loading Model's 
analytically-produced values for the same tasks. 
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IV. COMPARISON OF MIDAS MODEL DATA WITH STUDY DATA 



A. TASK LOADING MODEL WORKLOAD VALUES 

The MIDAS TLM generates workload data at a higher 
resolution than was measured by the survey forms. The TLM 
measures workload along four dimensions of human performance. 
These dimensions are the visual, auditory, cognitive, and 
motor activities performed by an aviator while flying. In its 
present configuration, the TLM averages workload values 
generated along each of the four dimensions for each of the 
individual and combined tasks. The tasks modelled in the TLM 
are comparable to the nine individual tasks and six combined 
tasks used in the pilot survey. The result is a set of task 
workload values as listed in Table 13. The numbers represent 
relative workload on a scale of 0 to 100, with larger numbers 
indicating higher workload. (Staveland, 1990, pp.6-9) 

Two different techniques were used with the TLM system to 
create workload values for the scout and attack helicopter 
pilot tasks. The first technique involved measuring only the 
workload generating attributes of the human performance 
dimensions involved in a task beyond the activity of flying. 
That is, only workload related to human performance above the 
basic activity of flying was calculated. This technique 
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generated the first two columns of data in Table 13 (ATK and 
SCT) . The second technique combined the workload attributes 
of a task with an underlying activity of basic flying. Thus 

TABLE 13. TLM-6ENERATED WORKLOAD VALUES 



INDIVIDUAL TASKS 


Group 

Code 


ATK 


SCT 


BSCATK 


BSCSCT 


Task 

No. 


Attack 

Pilots 


Scout 

Pilots 


Attack 

Pilots 

(Basic) 


Scout 

Pilots 

(Basic) 


1 


29 


29 


45 


45 


2 


27 


27 


42 


42 


3 


27 


27 


42 


42 


4 


26 


26 


42 


42 


5 


18 


18 


44 


44 


6 


26 


26 


42 


42 


7 


16 


13 


47 


46 


8 


19 


34 


36 


54 


9 


33 


32 


30 


46 


COMBINED TASKS 


1 


30 


30 


46 


46 


2 


60 


60 


42 


42 


3 


57 


57 


44 


44 


4 


47 


47 


46 


46 


5 


45 


39 


42 


50 


6 


56 


53 


52 


53 
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the workload of each task included the supposition of an 
ongoing basic hands-on flying activity. This technique 
generated the last two columns of data in Table 13 (BSCATK and 
BSCSCT) . (Staveland, 1990, pp. 19-23) 

From a visual inspection of the data in Table 13, it 
appears that the TLM-generated values for attack and scout 
pilots are similar in columns ATK and SCT and in columns 
BSCATK and BSCSCT, respectively. Except for tasks 7 and 8, 
the TLM values indicate that the scout and attack helicopter 
pilots should experience almost identical levels of workload 
for a given task. An additional observation is that many 
tasks within a column of data have identical or near identical 
workload values. For example, the individual-task portion of 
column BSCSCT contains four workload measurements of 42, two 
of 46, and one each of 44 and 45. According to the TLM model 
results, eight out of nine tasks involve essentially the same 
amount of workload. 

B. RANK CORRELATION OF MODEL DATA WITH SURVEY DATA 

The TLM model-generated workload values were compared with 
the survey workload values in a procedure similar to that 
described in Chapter III, Section G, where the task ratings 
were correlated with the paired comparisons using the Spearman 
rank correlation technique. The survey workload values used 
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for this correlation are from Tables 4, 5, 10, and 11. The 
model data used is all from Table 13. 

The results of the Spearman rank correlation technique are 
provided in Tables 14 to 17. Each of the first four tables 
corresponds to one of the four sections of the survey. Tables 
18 and 19 show the results of correlating the TLM with itself. 
Significance levels below 0.05, printed in bold type, indicate 
high correlation. 



TABLE 14. RANK CORRELATION SIGNIFICANCE LEVELS OF MODEL 
VERSUS STUDY WORKLOAD VALUES, FOR ATTACK 
HELICOPTER INDIVIDUAL TASKS 



TLM 

Group 

Code 


Survey Group Code 


AHR 


AHP 


ATK 


0.183 


0.358 


BSCATK 


0.403 


0.408 



TABLE 15. RANK CORRELATION SIGNIFICANCE LEVELS OF MODEL 
VERSUS STUDY WORKLOAD VALUES, FOR SCOUT 
HELICOPTER INDIVIDUAL TASKS 



TLM 


Survey Group Code 


Group 

code 

1 


SH-AB 

(ratings) 


SH-CV 

(ratings) 


SH-AB 

(comparisons) 


SH-CV 

(comparisons) 


SCT 


0.027 


0.521 


0.049 


0.225 


BSCSCT 


0.628 


0.504 


0.852 


0.604 
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The following observations may be made from the six tables: 

1. No significant correlations were found for the attack 
helicopter individual or combined tasks (Tables 14 and 
16) nor the scout helicopter combined tasks (Table 17) . 

2. Two significant correlations were found for the scout 
helicopter individual tasks (Table 15). The TLM's scout 
workload values were strongly correlated with both the 
ratings and paired-comparison values from the Attack 
Battalion scout pilots' (SH-AB) . However, only six scout 
pilots contributed to these two categories of survey 
data. 



TABLE 16. RANK CORRELATION SIGNIFICANCE LEVELS OF MODEL 
VERSUS STUDY WORKLOAD VALUES, FOR ATTACK 
HELICOPTER COMBINED TASKS 



TLM 

1 Group 
1 Code 


Survey Group Code 


AH-AB 

(ratings) 


AH-CV 

(ratings) 


AH-AB 

(comparisons) 


AH-CV 

(comparisons 


ATK 


0.565 


1.000 


0.482 


0.949 


BSCATK 


1.000 


0.841 


0.844 


1.000 



TABLE 17. RANK CORRELATION SIGNIFICANCE LEVELS OF MODEL 
VERSUS STUDY WORKLOAD VALUES, FOR SCOUT 
HELICOPTER COMBINED TASKS 



TLM 

Group 

Code 


Survey Group Code 


SHR 


SHP 


SOR-AB 


SOR-CV 


SOP-AB 


SCT 


0.655 


0.897 


0.565 


0.655 


0.565 


BSCSCT 


0.604 


0.490 


0.476 


0.604 


0.476 
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3. The significance levels for TLM-survey data correlations 
for both the scout and attack helicopter combined tasks 
(Tables 16 and 17) were generally poorer than those for 
the individual tasks (Tables 14 and 15) . This indicates 
that the workload results obtained via both technigues 
for individual tasks correlated somewhat better than did 
those for the combined tasks. 

4. The significance level values in Tables 18 and 19 
indicate that the TLM's two workload measurement 
techniques do not correlate highly with each other. 



TABLE 18. RANK CORRELATION SIGNIFICANCE LEVELS OF TLM 
INDIVIDUAL TASK WORKLOAD VALUES 



TLM 

Group 

Code 


TLM Group Code 


BSCATK 


BSCSCT 


ATK 


0.233 


- 


SCT 


- 


0.307 



TABLE 19. RANK CORRELATION SIGNIFICANCE LEVELS OF TLM 
COMBINED TASK WORKLOAD VALUES 



TLM 

Group 

Code 


TLM Group Code 


BSCATK 


BSCSCT 


ATK 


0.554 


- 


1 

SCT 


- 


0.218 
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C. GRAPHIC ANALYSIS OF MODEL DATA AMD SURVEY DATA 
1. Data Selection for Further Analysis 

Several graphic plots were prepared to make further 
comparisons between results from the two workload measuring 
devices. All of the survey data used for the graphic analysis 
came from the workload task ratings data, based on a scale of 
0 to 10. With a minor transformation, these values were 
directly comparable to the TLM output, which was based on a 
scale of 0 to 100. Survey data used for the graphic 
comparisons included AHR (Table 10, top) for the attack 
helicopter individual tasks, SH-CV (Table 4, top) for the 
scout helicopter individual tasks, AH-AB (Table 4, bottom) for 
the attack helicopter combined tasks, and SHR (Table 10, 
bottom) for the scout helicopter combined tasks. 

The four task-rating data vectors utilized for the 
graphic comparisons (AHR, SH-CV, AH-AB, and SHR) have been 
shown to be robust and internally valid for use as baseline 
workload values (Chapter III, Section G) . An additional 
criterion for the survey data used for the comparisons was 
that these vectors contained workload values from the greatest 
number of respondents for a given survey section. For 
example, the SH-CV vector (Table 4, top) was chosen over SH-AB 
(Table 4, top) because SH-CV represented responses from 12 
participants while SH-AB values were obtained from only six 
participants . 
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2. Results of Analysis 

The nine workload plots included in this section are 
based on a workload scale ranging from 0 to 100. Smaller 
values represent lower workload values. Survey data is 
plotted with a solid line in each graph. Model-generated data 
is plotted with a dashed line. Due to its demonstrated 
validity and robustness, the survey data has been taken as 
"ground truth" for these comparisons. 

The first plot, Figure 9, shows the only set of model- 
generated (SCT) and survey (SH-AB) workload values that 
correlated with a significance level below 0.05. Distinct 
differences in workload across the tasks may be observed. The 
model data generally tracks values from the survey data; 
however, the model data's "flatness" implies that it may have 
poorer resolution in measuring workload levels. Nevertheless, 
considering the high data correlation. Figure 9 serves as a 
base from which to view the other graphs. 

The remaining eight graphs are characterized by the 
following observations: 

1. Figure 10 shows that the attack helicopter individual- 
task model data (ATK) agrees reasonably well with the 
associated survey data (AHR) . Primary differences are 
for Task 1 (Follow scout) and Task 7 (Listen to radio 
transmission ) . 
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Figure 9. Highly Correlated Scout Helicopter Model (SCT) 
and Survey (SH-AB) Workload Values 




Figure 10. Attack Helicopter Model (ATK) and Survey (AHR) 
Workload Values 



2. Figure 11 indicates that the TLM BSCATK data over- 
estimates workload and provides little resolution for 
attack helicopter individual tasks. 
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Figure 11. Attack Helicopter Model (BSCATK) and Survey 
(AHR) Workload Values 



3. Figure 12 is very similar to Figure 9. The survey 
data of Figure 12 comes from the Cavalry Squadron 
scout pilots (SH-CV) . The survey data of Figure 9 comes 
from the Attack Battalion scout pilots (SH-AB) . 
Comparing Figure 9 with Figure 12, it may be seen that 
the model provides a slightly better measure of Attack 
Battalion scout pilot workload than Cavalry Squadron 
workload. The major discrepancy is for Task 7 (Listen 
to radio transmission) . 




Figure 12. Scout Helicopter Model (SCT) and Survey (SH- 
CV) Workload Values 
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4. Figure 13 is similar to Figure 11. Again, the BSCSCT 
data overestimates workload and provides little 
resolution for scout helicopter individual tasks. 




Figure 13. Scout Helicopter Model (BSCSCT) and Survey 
(SH-CV) Wor)cload Values 



5. Figures 14 through 17 all involve combined tasks and 
display similar results. In each case the model- 
generated workload values are much higher than the survey 
values for most tasks. The model values for ATK (Figure 
14) and SCT (Figure 16) show some trend agreement with 
the survey values, but greatly overestimate all but the 
first and last tasks. The model's BSCATK (Figure 15) and 
BSCSCT (Figure 17) values show almost no resolution of 
workload levels over the six combined tasks. 




COMBINED TASKS 



Figure 14. Attack Helicopter Model (ATK) and Survey (AH- 
AB) Workload Values 
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COMBINED TASKS 

Figure 15. Attack Helicopter Model (BSCATK) and Survey 
(AH-AB) Workload Values 




COMBINED TASKS 



Figure 16. Scout Helicopter Model (SCT) and Survey (SHR) 
Workload Values 
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WORKLOAD 




COMBINED TASKS 



Figure 17. Scout Helicopter Model (B8C8CT) and Survey 
(8HR) Workload Values 
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V. CONCLUSIONS AND RECOMMENDATIONS 



A. CONCLUSIONS 

The purpose of this study was to determine the validity of 
the A^I TLM as a device to measure the workload levels 
experienced by Army helicopter pilots. Empirically-derived 
workload data was collected from Army pilots, analyzed, and 
correlated with workload measurements generated by the TLM. 
Analysis of the empirical data provided insights into pilot- 
generated workload levels and provided a statistically 
acceptable basis upon which to judge the accuracy of TLM 
outputs. Furthermore, insights into the opinions of Army 
helicopter pilots concerning workload provides direction for 
future refinement of the TLM. 

1. TLM Validity 

In its present configuration, output from the TLM does 
not consistently yield measurements of Army helicopter pilot 
workload that correlate with pilot opinions of their workload 
levels, for the set of tasks tested. Of the two TLM 
techniques of evaluating workload, the second technique should 
be abandoned. This technique adds task-generated workload 
onto constant workload resulting from a basic underlying 
flying task (BSCATK and BSCSCT) . This technique showed little 
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workload level resolution and overestimated workload values 



for almost every task. 

However, the first technique utilized by the TLM 
appears to provide better results. This technique analyzes 
aviator tasks separately from the actions associated with 
hands-on flying. Graphic analysis showed workload trends that 
in part agreed with the empirical measurements of workload. 
However, it should be noted that agreement between the model 
and the survey data was greater for the individual task 
measurements than for those related to the combined tasks. 

2. Army Helicopter Pilot Workload Perceptions 

Outputs from the TLM currently indicate that workload 
levels for the scout and attack helicopter pilots are almost 
identical. For essentially the same set of tasks embedded in 
realistic scout and attack helicopter scenarios, the data 
collected via the pilot surveys indicates that aviator 
workload varies according to the type aircraft flown and the 
type of mission conducted by the pilots. It is obvious that 
there are major discrepancies between the two methods for 
determining workload. 

Results of the Friedman test indicate that survey 
responses from the Attack Helicopter Battalion participants 
cannot be pooled with those from the Cavalry Squadron. This 



56 



is a strong indication that workload levels are not 
experienced the same by all of the aviators tested. The 
Spearman and Wilcoxon tests showed that, even among the scout 
pilots, workload is perceived differently by those scouts in 
the Cavalry Squadron than by those in the Attack Helicopter 
Battalion. These results provide strong evidence that the TLM 
may need modification to adjust workload according to the 
aviator's aircraft and specific mission. 

B. RECOMMENDATIONS 

Analysis of the degree of correlation between the TLM's 
workload output and the empirically-generated workload values 
obtained via the survey results in three recommendations. 
First, the TLM in its present configuration should not be used 
as the workload measuring mechanism in the MIDAS program. The 
TLM generally overestimates workload and provides an overly- 
narrow range of values for workload experienced by pilots 
performing a variety of tasks. 

Second, the TLM shows promising potential in the measuring 
of workload associated with individual helicopter pilot tasks. 
This aspect of the TLM should be enhanced to take maximum 
advantage of this capability. Initial enhancements should 
address the mission and aircraft involved in generating pilot 
workload. Although the survey data indicates that these 
factors are significant, the TLM generates generic workload 
measurements independent of them. 
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Third, empirical workload data collected as described in 
this thesis should be used for future evaluations of the TLM. 
As the TLM matures, efforts should be made to ensure that 
output correlates highly with workload data obtained directly 
from Army pilots, as described in this study. However, the 
limited range of the survey data reported here should not 
serve as the only means of validating the TLM. TLM output 
should also be compared to other empirically-derived workload 
measurements, and to the output of other models designed to 
measure pilot workload. 

C. SUGGESTIONS FOR FURTHER RESEARCH 

Several aspects of this study deserve further invest- 
igation. First, helicopter pilot workload data should be 
obtained from another, independent source. This data then 
could be compared with the empirically-derived data collected 
in this study, to insure data validity. This same independent 
workload data also could be compared to the TLM's output. 
Second, the survey results reported here may be based on 
factors not investigated in this study, such as individual 
pilot experience. A more detailed analysis of the individual 
pilot responses may be warranted. Third, expanded surveys 
similar to those used for this study should be conducted in 
the future to broaden and refine the workload data base. 
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APPENDIX A. SCOUT HELICOPTER WORKLOAD SURVEY 



SURVEY ON HELICOPTER PILOTING WORKLOAD 



Thank you for your participation in this survey. Your 
assistance helps a NASA-Army team develop a human factors 
design tool aimed at improving the layout of future Army 
helicopter cockpits. 

There are no right or wrong answers to the questions in 
this survey. The intent is to determine relative workload 
that you experience when flying. Your responses will help us 
determine what tasks you feel are harder or easier than others 
for the given scenario. Please answer the questions candidly. 

The survey is anonymous. The combined data from all of 
the surveys will produce an overall "pilot” profile of task 
ratings . 

The ratings and comparisons in the survey are listed 
randomly. No two surveys are exactly alike. 

If you have any questions, hold up your hand and the 
survey administrating officer will assist you. 



PAGE 1 OF 12 
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HOW TO FILL OUT THE SURVEY 



Evaluate the following maneuvers as an individual crewman 
in the context of the given scenario and helicopter terrain 
flight in general. Difficulty is defined as a relative measure 
of combined physical and mental effort. 

Evaluations are done in two parts. First, rate each 
maneuver by circling a value along the degree-of-dif f iculty 
continuum to indicate its difficulty. 



Example 1: 



MANEUVER 



DEGREE OF DIFFICULTY 



low 






high 


Fly straight and level 0123 


4 


1 

CO 

1 

1 

VO 

in 

1 


9 10 


Autorotate at night 0123 


4 


1 

CO 

1 

1 

VO 

1 

in 

1 


9 10 


For the second part of the survey 
are compared with each other. 


, individual maneuvers 


Choose the more difficult of the 
circle next to it. 


two 


by filling 


in the 


Example 2 : 








Fly straight and level 0 0 


Autorotate at 


night 






PAGE 


2 OF 12 
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SCOUT HELICOPTER SCENARIO 



SETTING: You are the pilot of an OH-58 conducting a route 
reconnaissance in an unfamiliar hilly/wooded area. You are in 
the terrain flight mode along one side of the route, clearing 
the area while observing the route as frequently as possible. 
Due to pilot shortages, you are accompanied by a crewchief 
sitting in the left seat. He is familiar with the cockpit. 
He can hold a 1:50,000 map for you to observe. However, he 
cannot fly and you must double check his map reading 
activities. You are flying during the day with clear and calm 
weather conditions. An attack helicopter is overwatching you. 
However, you will not communicate with him during this mission 
segment. Enemy contact is possible. 



PAGE 3 OF 12 
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Rate the difficulty of each maneuver by circling a value along 
the degree of difficulty continuum. 



MANEUVER 



DEGREE OF DIFFICULTY 



low 



high 



Follow another scout 0123456789 10 



NOEdecelerat ion 0 123456789 10 



Check torque meter and announce reading 0 1 2 3 4 5 6 7 8 9 10 



Hover 0123456789 10 



Tell crewchief "Unmasking” 0 1 2 3 4 5 6 7 8 9 10 



Unmaskaircraf 1 0 123456789 10 



Listen to crewchief describe what he sees 0123456789 10 



Turn up volume on ICS control panel 0 1 2 3 4 5 6 7 8 9 10 



Determine present location (6 digit grid) 0 1 2 3 4 5 6 7 8 9 10 

as crewchief holds map 



SURVEY # OHITR 1 
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Rate the difficulty of each maneuver by circling a value along 
the degree of difficulty continuum. 

MANEUVER DEGREE OF DIFFICULTY 



low 



high 



NOE deceleration behind another scout 0123456789 10 



Check torque meter and announce reading 0 1 2 3 4 5 6 7 8 9 10 

while NOE decelerating 



Check torque meter and announce reading 0 1 2 3 4 5 6 7 8 9 10 

while hovering 



Tell crewchief "Unmasking" while unmasking 0 1 2 3 4 5 6 7 8 9 10 

aircraft 



Listen to crewchief describe what he sees 0 1 2 3 4 5 6 7 8 9 10 

while you turn up volume on ICS control 

panel and hover 



Determine present location (6 digit grid) 0 1 2 3 4 5 6 7 8 9 10 

as crewchief holds map and you listen 
to him describe what he sees 



SURVEY # OHCTR 



1 
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Compare the two maneuvers on the same line. Choose the more 
difficult of the two by filling in the circle next to it. 

MANEUVER 1 MANEUVER 2 



Follow another scout O O NOE deceleration 



Follow another scout O O Check torque meter and announce 

reading 



Follow another scout O O Hover 



Follow another scout O O Tell crewchief ’’Unmasking” 



Follow another scout O O Unmask aircraft 



Follow another scout O O Listen to crewchief describe what 

he sees 



Follow another scout O O Turn up volume on ICS control panel 



Follow another scout O O Determine present location (6 digit 

grid) as crewchief holds map 



NOE deceleration O O Check torque meter and announce 

reading 



NOE deceleration O O Hover 



NOE deceleration O O Tell crewchief ’’Unmasking” 



NOE deceleration O O Unmask aircraft 
SURVEY # OHIPC 1-1 PAGE 6 OF 12 
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MANEUVER 1 



MANEUVER 2 







NOE 


deceleration O 


O Listen to crewchief describe what 
he sees 






NOE 


deceleration O 


O Turn up volume on ICS control panel 






NOE 


deceleration O 


O Determine present location (6 digit 
grid) as crewchief holds map 


Check 


torque 


meter 


and announce O 
reading 


O Hover 


Check 


torque 


meter 


and announce O 
reading 


O Tell crewchief "Unmasking" 


Check 


torque 


meter 


and announce O 
reading 


O Unmask aircraft 


Check 


torque 


meter 


and announce O 
reading 


O Listen to crewchief describe what 
he sees 


Check torque 


meter 


and announce O 
reading 


O Turn up volume on ICS control panel 


Check torque meter 


and announce O 
reading 


O Determine present location (6 digit 
grid) as crewchief holds map 








Hover O 


O Tell crewchief "Unmasking" 








Hover O 


O Unmask aircraft 



SURVEY # OHIPC 1-2 
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MANEUVER 1 



MANEUVER 2 



Hover O O Listen to crewchief describe what 
he sees 



Hover O O Turn up volume on ICS control panel 



Hover O O Determine present location (6 digit 
grid) as crewchief holds map 



Tell crewchief "Unmasking” O 



Tell crewchief "Unmasking" O 



O Unmask aircraft 



O Listen to crewchief describe what 
he sees 



Tell crewchief "Unmasking" O O Turn up volume on ICS control panel 



Tell crewchief "Unmasking" O O Determine present location (6 digit 

grid) as crewchief holds map 



Unmask aircraft O O Listen to crewchief describe what 

he sees 



Unmask aircraft O O Turn up volume on ICS control panel 



Unmask aircraft O O Determine present location (6 digit 

grid) as crewchief holds map 



Listen to crewchief describe what O O Turn up volume on ICS control panel 

he sees 



SURVEY # OHIPC 1-3 
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MANEUVER 1 



MANEUVER 2 



Listen to crewchief describe what O 

he sees 


O Determine present location (6 digit 
grid) as crewchief holds map 


Turn up volume on ICS control panel O 


O Determine present location (6 digit 
grid) as crewchief holds map 



SURVEY # OHIPC 1-4 
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Compare the two maneuvers on the same line. Choose the more 
difficult of the two by filling in the circle next to it. 



MANEUVER 1 



MANEUVER 2 



NOE deceleration behind another O O Check torque meter and announce 

scout reading while NOE decelerating 



NOE deceleration behind another O O Check torque meter and announce 

scout reading while hovering 



NOE deceleration behind another O O Tell crewchief "Unmasking” while 

scout unmasking aircraft 



NOE deceleration behind another O O Listen to crewchief describe what 

scout he sees while you turn up volume 

on ICS control panel and hover 



NOE deceleration behind another O O Determine present location (6 digit 

scout grid) as crewchief holds map and 

you listen to him describe what 
he sees 



Check torque meter and announce O O Check torque meter and announce 
reading while NOE decelerating reading while hovering 



Check torque meter and announce O O Tell crewchief "Unmasking" while 
reading while NOE decelerating unmasking aircraft 



Check torque meter and announce O O Listen to crewchief describe what 
reading while NOE decelerating he sees while you turn up volume 

on ICS control panel and hover 



SURVEY # OHCPC 1-1 
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MANEUVER 1 



MANEUVER 2 



Check torque meter and announce O O Determine present location (6 digit 
reading while NOE decelerating grid) as crewchief holds map and 

you listen to him describe what 
he sees 



Check torque meter and announce O O Tell crewchief "Unmasking" while 
reading while hovering unmasking aircraft 



Check torque meter and announce O O Listen to crewchief describe what 
reading while hovering he sees while you turn up volume 

on ICS control panel and hover 



Check torque meter and announce O O Determine present location (6 digit 
reading while hovering grid) as crewchief holds map and 

you listen to him describe what 
he sees 



Tell crewchief "Unmasking" while O O Listen to crewchief describe what 
unmasking aircraft he sees while you turn up volume 

on ICS control panel and hover 



Tell crewchief "Unmasking" while O O Determine present location (6 digit 
unmasking aircraft grid) as crewchief holds map and 

you listen to him describe what 
he sees 



Listen to crewchief describe what O 
he sees while you turn up volume 
on ICS control panel and hover 



O Determine present location (6 digit 
grid) as crewchief holds map and 
you listen to him describe what 
he sees 



SURVEY # OHCPC 1-2 
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PILOT BACKGROUND INFORMATION 



Please answer the following questions regarding your 
aviation experience and qualifications. 

1. Approximately how many flight hours do you have in 
the following aircraft as either a PC or PI? 

(Leave blank if little or no experience) 

0H-58A/C AH-1 

0H-58D AH-64 

UH-1 other rotary wing 

UH-60 fixed wing 

2. Answer the following by indicating the approximate 
number of flight hours . 

a) Your total flight time: 

b) Flight time as an Instructor Pilot: 

c) Combat flight time: 

d) Flight time while qualified as a PC: 

3. Answer the following by indicating the approximate 
number of years . 

a) Years as a PC: 

b) Years as an Army Aviator: 



END OF SURVEY 

THANK YOU FOR YOUR TIME AND EFFORT 
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APPENDIX B. ATTACK HELICOPTER WORKLOAD SURVEY 



SURVEY ON HELICOPTER PILOTING WORKLOAD 



Thank you for your participation in this survey. Your 
assistance helps a NASA-Army team develop a human factors 
design tool aimed at improving the layout of future Army 
helicopter cockpits. 

There are no right or wrong answers to the guestions in 
this survey. The intent is to determine relative workload 
that you experience when flying. Your responses will help us 
determine what tasks you feel are harder or easier than others 
for the given scenario. Please answer the guestions candidly. 

The survey is anonymous. The combined data from all of 
the surveys will produce an overall "pilot" profile of task 
ratings . 

The ratings and comparisons in the survey are listed 
randomly. No two surveys are exactly alike. 

If you have any guestions, hold up your hand and the 
survey administrating officer will assist you. 
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HOW TO FILL OUT THE SURVEY 



Evaluate the following maneuvers as an individual crewman 
in the context of the given scenario and helicopter terrain 
flight in general. Difficulty is defined as a relative measure 
of combined physical and mental effort. 

Evaluations are done in two parts. First, rate each 
maneuver by circling a value along the degree-of-dif f iculty 
continuum to indicate its difficulty. 



Example 1: 



MANEUVER 



DEGREE OF DIFFICULTY 



low 






high 


Fly straight and level 0123 


4 


1 

(J1 

1 

C\ 

1 

1 

00 

1 


9 10 


Autorotate at night 0123 


4 


5 6 7 8 


9 10 


For the second part of the survey 
are compared with each other. 


, individual maneuvers 


Choose the more difficult of the 
circle next to it. 


two 


by filling 


in the 


Example 2 : 








Fly straight and level 0 0 


Autorotate at 


night 






PAGE 


2 OF 12 
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ATTACK HELICOPTER SCENARIO 



SETTING: You are the pilot of an armed AH-1 flying in the 
terrain flight mode. You are conducting travelling overwatch 
by following an aeroscout through an unfamiliar hilly/wooded 
area to a battle position. You will receive a target handoff 
from the firing position. Your crewmember will prepare for a 
TOW missile engagement and will not fly the aircraft during 
this mission segment. You are flying during the day with 
clear and calm weather conditions. Enemy contact is possible. 



PAGE 3 OF 12 
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Rate the difficulty of each maneuver by circling a value along 
the degree of difficulty continuum. 



MANEUVER 



DEGREE OF DIFFICULTY 



low 



high 



FollowscoutO 123456789 10 



NOEdecelerationO 123456789 10 



Check torque meter and announce reading 0 1 2 3 4 5 6 7 8 9 10 



Hover 0123456789 10 



Tell crewmember "Unmasking" 0 1 2 3 4 5 6 7 8 9 10 



Unmaskaircraft 0 123456789 10 



Listen to radio transmission from scout 0 1 2 3 4 5 6 7 8 9 10 

(direction to fire^ range, target) 



Switch master arm to "arm" 0123456789 10 



Interpret gunners position of TSU based on 0 1 2 3 4 5 6 7 8 9 10 

PSI information 



SURVEY # AHITR 1 
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Rate the difficulty of each maneuver by circling a value along 
the degree of difficulty continuum. 



MANEUVER 



DEGREE OF DIFFICULTY 



low 



high 



NOE deceleration behind scout 0123456789 10 



Check torque meter and announce reading 0 1 2 3 4 5 6 7 8 9 10 

while NOE decelerating 



Check torque meter and announce reading 0 1 2 3 4 5 6 7 8 9 10 

while hovering 



Tell crewmember "Unmasking" while unmasking 0 1 2 3 4 5 6 7 8 9 10 

aircraft 



Listen to radio transmission from scout 0 1 2 3 4 5 6 7 8 9 10 

(direction to fire, range, target) while 
switching master arm to "arm" 



Interpret gunners position of TSU, based 0 1 2 3 4 5 6 7 8 9 10 

on PSI information, while receiving 
target handoff from scout 



SURVEY # AHCTR 1 
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Compare the two maneuvers on the same line. Choose the more 
difficult of the two by filling in the circle next to it. 

MANEUVER 1 MANEUVER 2 





Follow 


scout 


o 


O 


NOE deceleration 




Follow 


scout 


o 


O 


Check torque meter and announce 
reading 




Follow 


scout 


o 


O 


Hover 




Follow 


scout 


o 


O 


Tell crewmember "Unmasking" 




Follow 


scout 


o 


O 


Unmask aircraft 




Follow 


scout 


O 


O Listen to radio transmission from 
scout (target, range, direction) 




Follow 


scout 


O 


O 


Switch master arm to "arm" 




Follow 


scout 


O 


O Interpret gunners position of TSU 
based on PSI information 


NOE 


deceleration 


O 


O 


Check torque meter and announce 
reading 


NOE 


deceleration 


0 


0 


Hover 


NOE 


deceleration 


O 


O 


Tell crewmember "Unmasking" 



SURVEY # AHIPC 1-1 
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MANEUVER 1 



MANEUVER 2 



NOE deceleration O 


O Unmask aircraft 


NOE deceleration O 


O Listen to radio transmission from 
scout (target, range, direction) 



NOE deceleration O O Switch master arm to ’’arm” 



NOE deceleration O O Interpret gunners position of TSU 

based on PSI information 



Check torque meter and announce O O Hover 

reading 



Check torque meter and announce O O Tell crewmember "Unmasking" 

reading 



Check torque meter and announce O O Unmask aircraft 

reading 



Check torque meter and announce O O Listen to radio transmission from 

reading scout (target, range, direction) 



Check torque meter and announce O O Switch master arm to "arm" 

reading 



Check torque meter and announce O O Interpret gunners position of TSU 

reading based on PSI information 



Hover O O Tell crewmember "Unmasking" 



SURVEY # AHIPC 1-2 
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MANEUVER 1 



MANEUVER 2 





Hover O 


O Unmask aircraft 




Hover O 


O Listen to radio transmission from 
scout (target, range, direction) 




Hover O 


O Switch master arm to "arm" 




Hover O 


O Interpret gunners position of TSU 
based on PSI information 


Tell 


crewmember "Unmasking" 0 


O Unmask aircraft 


Tell 


crewmember "Unmasking" O 


O Listen to radio transmission from 
scout (target, range, direction) 


Tell 


crewmember "Unmasking" O 


O Switch master arm to "arm" 


Tell 


crewmember "Unmasking" O 


O Interpret gunners position of TSU 
based on PSI information 




Unmask aircraft O 


O Listen to radio transmission from 
scout (target, range, direction) 




Unmask aircraft O 


0 Switch master arm to "arm" 




Unmask aircraft 0 


O Interpret gunners position of TSU 
based on PSI information 



Listen to radio transmission from O O Switch master arm to "arm" 
scout (target, range, direction) 



SURVEY # AHIPC 1-3 
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MANEUVER 1 



MANEUVER 2 



Listen to radio transmission from O O Interpret gunners position of TSU 
scout (target, range, direction) based on PSI information 



Switch master arm to "arm” O O Interpret gunners position of TSU 

based on PSI information 



SURVEY # AHIPC 1-4 
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Compare the two maneuvers on the same line. Choose the more 
difficult of the two by filling in the circle next to it. 



MANEUVER 1 


MANEUVER 2 



NOE deceleration behind scout O O Check torque meter and announce 

reading while NOE decelerating 



NOE deceleration behind scout O O Check torque meter and announce 





reading while hovering 


NOE deceleration behind scout O 


O Tell crewmember "Unmasking" while 
unmasking aircraft 



NOE deceleration behind scout O O Listen to radio xsmn from scout 





(direction, range, target) while 
switching master arm to "arm" 


NOE deceleration behind scout O 


O Interpret gunners position of TSU 
based on PSI info, while 
receiving target handoff from 
scout 



Check torque meter and announce O O Check torque meter and announce 



reading while NOE decelerating 


reading while hovering 



Check torque meter and announce O O Tell crewmember "Unmasking" while 



reading while NOE decelerating 


unmasking aircraft 



Check torque meter and announce O O Listen to radio xsmn from scout 



reading while NOE decelerating 


(direction, range, target) while 
switching master arm to "arm" 


Check torque meter and announce O 
reading while NOE decelerating 


O Interpret gunners position of TSU 
based on PSI info, while 
receiving target handoff from 
scout 


SURVEY # AHCPC 1-1 
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MANEUVER 1 



MANEUVER 2 



Check torque meter and announce O O Tell crev/member "Unmasking" while 
reading while hovering unmasking aircraft 



Check torque meter and announce O O Listen to radio xsmn from scout 
reading while hovering (direction, range, target) while 

switching master arm to "arm" 



Check torque meter and announce O O Interpret gunners position of TSU 
reading while hovering based on PSI info, while 

receiving target handoff from 
scout 



Tell crewmember "Unmasking" while O O Listen to radio xsmn from scout 
unmasking aircraft (direction, range, target) while 

switching master arm to "arm" 



Tell crewmember "Unmasking" while O O Interpret gunners position of TSU 
unmasking aircraft based on PSI info, while 

receiving target handoff from 
scout 



Listen to radio xsmn from scout O O Interpret gunners position of TSU 
(direction, range, target) while based on PSI info, while 

switching master arm to "arm" receiving target handoff from 

scout 



SURVEY # AHCPC 1-2 
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PILOT BACKGROUND INFORMATION 



Please answer the following questions regarding your 
aviation experience and qualifications. 

1. Approximately how many flight hours do you have in 
the following aircraft as either a PC or PI? 

(Leave blank if little or no experience) 

OH-58A/C AH-1 

OH-58D AH-64 

UH-1 other rotary wing 

UH-60 fixed wing 

2 . Answer the following by indicating the approximate 
number of flight hours . 

a) Your total flight time: 

b) Flight time as an Instructor Pilot: 

c) Combat flight time: 

d) Flight time while qualified as a PC: 

3. Answer the following by indicating the approximate 
number of years . 

a) Years as a PC: 

b) Years as an Army Aviator: 



END OF SURVEY 

THANK YOU FOR YOUR TIME AND EFFORT 
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APPENDIX C. SPEJ^RMAN RANK CORRELATION SIGNIFICANCE LEVELS 



INDIVIDUAL-TASK PAIREl 


D COMPARISONS 




AH-AB 


AH-CV 


SH-AB 


SH-CV 


EO-AB 


EO-CV 


AH-AB 


- 


0.006 


- 


- 


- 


- 


AH-CV 




- 


- 


- 


- 


- 


SH-AB 






- 


0.011 


0.009 


- 


SH-CV 








- 


- 


0.048 


EO-AB 










- 


0.013 


EO-CV 












- 



INDIVIDUAL-TASK RATINGS 




AH-AB 


AH-CV 


SH-AB 


SH-CV 


EO-AB 


EO-CV 


AH-AB 


- 


0.018 


- 


- 


- 


- 


AH-CV 




- 


- 


- 


- 


- 


SH-AB 






- 


0.079 


0.069 


- 


SH-CV 








- 


- 


0.027 


EO-AB 










- 


0.011 


EO-CV 












- 
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COMBINED-TASK PAIRED COMPARISONS 




AH-AB 


AH-CV 


SH-AB 


SH-CV 


EO-AB 


EO-CV , 


AH-AB 


- 


0.110 


- 


- 


- 


- 


AH-CV 




- 


- 


- 




- 


SH-AB 






- 


0.000 


0.000 


- 


SH-CV 








- 


- 


0.060 


EO-AB 










- 


0.060 


EO-CV 












- 



COMBINED-TASK RATINGS 




AH-AB 


AH-CV 


SH-AB 


SH-CV 


EO-AB 


EO-CV 


AH-AB 


- 


0.136 


- 


- 


- 


— 


AH-CV 




- 


- 


- 


- 


- 


SH-AB 






- 


0.035 


0.035 


- 


SH-CV 








- 


- 


0.035 


EO-AB 










- 


0.035 


EO-CV 












- 



84 



LIST OF REFERENCES 



Army-NASA Aircrew Aircraft Integration Program, Executive 
Summary, NASA Ames Research Center, 1990. 

Chambers, J. M. , and others. Graphical Methods for Data 
Analysis, pp. 82-86, Wadsworth & Brooks, 1983. 

Dunn-Rankin, P., Scaling Methods , pp. 79-82, Lawrence Erlbaum 
Associates, 1983. 

Hamburg, M., Statistical Analysis for Decision Making, pp. 
576-578, Harcourt Brace Jovanovich, 1987. 

Hart, S. G., and Staveland, L. E. , Development of NASA-TLX 
(Task Load Index) : Results of Empirical and Theoretical 

Research, NASA-Ames Research Center, 1988. 

Kantowitz, B. H., and Casper, P. A., "Human Workload in 
Aviation," Human Factors in Aviation, Academic Press, 1988. 

Mendenhall, W. , Scheaffer, R. L. , and Wackerly, D. D. , 
Mathematical Statistics with Applications, 4th ed. , PWS-Kent 
Publishing Co., 1990. 

Rice, J. A., Mathematical Statistics and Data Analysis, 
pp. 336-337, Wadsworth & Brooks, 1988. 

Rosenthal, R., and Rosnow, R.L., Essentials of Behavioral 
Research : Methods and Data Analysis, pp. 81-139, McGraw-Hill 
Book Co. , 1984 . 

Staveland, L., Army-NASA Aircrew /Aircraft Integration Program 
Task Loading Model Software Documentation, NASA-Ames Research 
Center, 1990. 

Thurstone, L. L. , The Measurement of Values, pp. 67-81, The 
University of Chicago Press, 1963. 

United States Army Research Institute for the Behavioral and 
Social Sciences TR 851, Operator Workload : Comprehensive 

Review and Evaluation of Operator Workload Methodologies , by 
R. J. Lysaght and others, pp. 81-99, June 1989. 

Zatkin, J. L. , Scaling by Magnitude Estimation, Naval 
Postgraduate School, pp. 1-6, 1983. 



85 



INITIAL DISTRIBUTION LIST 



No. 



1. Defense Technical Information Center 
Cameron Station 

Alexandria, VA 22304-6145 

2. Library, Code 52 

Naval Postgraduate School 
Monterey, CA 93943-5002 

3 . NASA-Ames Research Center 
ATTN: Jim Hartzell 
MS262-1 

Moffett Field, CA 94035-1000 

4. Professor Judith Lind 
Code OR/Li 

Naval Postgraduate School 
Monterey, CA 93943-5000 

5. CDR Thomas Mitchell 
Code 602 

Naval Air Development Center 
Warminster, PA 18974-5000 

6. CPT James Galvin 
4 Wren Way Court 
Stafford, VA 22554 

7. Joseph Kern 
Code 6022 

Naval Air Development Center 
Warminster, PA 18974 



Copies 

2 

2 

4 

2 

2 

2 

1 



86 



Thesis 

G1417 Galvin 

c.l Correlational analysis 

of survey and model-gene- 
rated workload values. 



Thesis 

G1417 Galvin 

c.l Correlational analysis 

of survey and model-gene- 
rated workload values. 



